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Abstract 

NASA has been collecting Earth observation data for over 50 years using instruments on board satellites, 
aircraft and ground-based systems. With the inception of the Earth Observing System (EOS) Program in 
1990, NASA established the Earth Science Data and Information System (ESDIS) Project and initiated 
development of the Earth Observing System Data and Information System (EOSDIS). A set of Distributed 
Active Archive Centers (DAACs) was established at locations based on science discipline expertise. Today, 
EOSDIS consists of 12 DAACs and 12 Science Investigator-led Processing Systems (SIPS), processing data 
from the EOS missions, as well as the Suomi National Polar Orbiting Partnership mission, and other satellite 
and airborne missions. The DAACs archive and distribute the vast majority of data from NASA’s Earth 
science missions, with data holdings exceeding 1 2 petabytes The data held by EOSDIS are available to all 
users consistent with NASA’s free and open data policy, which has been in effect since 1 990. The EOSDIS 
archives consist of raw instrument data counts (level 0 data), as well as higher level standard products (e.g., 
geophysical parameters, products mapped to standard spatio-temporal grids, results of Earth system models 
using multi-instrument observations, and long time series of Earth System Data Records resulting from 
multiple satellite observations of a given type of phenomenon) . 

EOSDIS data stewardship responsibilities include ensuring that the data and information content are reliable, 
of high quality, easily accessible, and usable for as long as they are considered to be of value. 


1. INTRODUCTION 

The U.S. National Aeronautics and Space Administration (NASA) has been collecting data from Earth 
observations for over 50 years using spaceborne, airborne and ground-based instruments. Among the early 
spaceborne missions were the Nimbus series of satellites and the Landsat series, the latter of which 
continues to the present day. The Earth Observing System (EOS) Program was initiated by NASA in 1990, 
resulting in a series of satellite launches, starting with the Tropical Rainfall Measuring Mission (TRMM) in 
November 1997. This was followed by Landsat-7 and Terra (1999), Aqua (2002), ICESat (2003), Aura (2004) 
and other smaller satellites. Most of the EOS satellites continue to operate and provide a wealth of 
measurements of various parameters that characterize the Earth. Following the EOS Program, there are the 
Suomi National Polar-orbiting Partnership (SNPP), the Decadal Survey satellite missions and satellite 
missions that are part of the Earth Systems Science Pathfinder Program. Also, there are several aircraft 
campaigns constituting the Earth Venture Suborbital (EVS) Program. In addition, there are many field 
campaigns and validation projects supported by NASA that result in hundreds of data products. The data 
from all these sources constitute a valuable national resource, which is extremely useful for scientific 
research and applications on a global scale. In recognition of this, at the inception of the EOS Program, 
NASA established the ESDIS Project to develop and operate the EOSDIS. The ESDIS Project is responsible 
for archiving and distribution of most of NASA’s Earth science data. 

Today, EOSDIS, consists of 12 Earth Science-discipline based DAACs and 12 Science Investigator-led 
Processing Systems The data held by EOSDIS are available to all users, consistent with NASA’s free and 
open data policy, which has been in effect since 1990. The EOSDIS archives consist of raw instrument data 
counts (level 0 data), as well as higher level standard products (e.g., geophysical parameters, products 
mapped to standard spatio-temporal grids, results of Earth system models using multi-instrument 
observations, and long time series of Earth System Data Records resulting from multiple satellite 
observations of a given type of phenomenon). The EOSDIS has been distributing data to a broad, diverse 
and global user community since 1994. During 2014, the distribution from EOSDIS exceeded one billion files. 



The ESDIS Project’s data stewardship responsibilities include ensuring that the data and information content 
are reliable, of high quality, easily accessible, and usable for as long as they are considered to be of value. 
To meet these responsibilities, the ESDIS Project coordinates with the data producing missions or projects 
throughout their lifecycles, establishing Interface agreements and data management plans. The products are 
produced by science teams with peer-reviewed algorithms, quality assessed and documented with 
appropriate caveats about usage. The capabilities in EOSDIS ensure easy accessibility to data and 
associated services for discovery, visualization, and downloading for purposes of science and applications. 
Near real-time access is available to some of the data. NASA ensures that the active archive capabilities are 
available as long as there is interest in using the data, which is well beyond the life time of the missions. In 
201 1 , NASA developed the Earth Science Data Preservation Content Specifications document, which is 
being used to ensure that the data and all the associated metadata and documentation from missions are 
collected and archived in preparation for permanent preservation, so that future generations will still be able 
to understand and use the data products from today’s missions. 

The purpose of this paper is to describe the activities carried out by the ESDIS Project to meet its data 
stewardship responsibilities. Section 2 presents the different project lifecycles that need to be 
accommodated by the ESDIS Project. Section 3 discusses the types of agreements with the various projects 
that originate data archived and distributed by the EOSDIS. Section 4 provides an overview of the 
capabilities of EOSDIS. Section 5 presents the steps taken by the ESDIS Project to ensure persistence of 
data beyond the project lifetimes and summarizes the ESDIS Project’s interagency and international 
collaborations in the area of data stewardship. Section 6 provides a summary and conclusions. 


2. PROJECT LIFECYCLES 

As indicated above, NASA’s Earth science data originate from various types of missions/projects. The types 
and schedules of interactions needed to ensure that these data receive appropriate stewardship will depend 
on the lifecycles of the missions/projects. To illustrate this, we discuss lifecycles of and processes involved in 
satellite missions, which typically involve high cost and risk and are long-lived (5- to 6-year prime missions 
followed with extended operations). 

Requirements for managing satellite projects are detailed in the NASA Procedural Requirement (NPR) 

71 20. 5E (NASA, 201 2). Figure 1 , based on NPR 71 20.5E illustrates the various events involved in a single- 
satellite project’s lifecycle. The life-cycle starts with concept studies (Pre-Phase A) and involves several 
gates (Key Decision Points or KDP’s) as the project proceeds through formulation and implementation. While 
the details may vary from project to project, the following is a notional set of key events needed from the 
standpoint of managing Earth science data resulting from such satellite projects. 

• Pre-Phase A/Phase A (Concept Studies and Technology Development; through KDP-B): 1. Define 
science products and estimate capacity requirements; 2. Estimate costs for processing, archiving 
and distributing data and derived science products; 3. Assign one or more DAACs to archive and 
distribute data and derived products; 4. Develop Inter-Project Agreement between the ESDIS Project 
and the flight project. 

• Phase B (Preliminary Design & Technology Completion; through KDP-C): 1. Flight Project generates 
Science Data Management Plan (DMP) (Preliminary DMP delivered at KDP-C); 2. Science team’s 
science data processing group and DAACs develop Interface Requirements Document; 3. Science 
teams develop Algorithm Theoretical Basis Documents (ATBDs). 

• Phase C (Final Design & Fabrication; through KDP-D): 1. Science team’s science data processing 
group and DAACs develop Interface Control Documents (ICD); 2. Updates are made to Science 
DMP as needed. 

• Phase D (System Assembly, Integration & Test, Launch & Checkout; through KDP-E): 1. Science 
team’s science data processing group and DAACs develop Operations Agreement; 2. ESDIS 
Project, DAACs and science team’s science data processing group perform interface testing; 3. End- 



to-end mission operations and science systems tests are conducted; 4. Science DMP is baselined 
and updated as needed. 

• Phase E (Operations & Sustainment; through KDP-F): 1 . Science operations start and go on for the 
life of mission (design life plus extended operations); 2. Data, derived science products, associated 
metadata and documentation are collected by the DAACs on an on-going basis. 

• Phase F (Closeout): All contents needed to preserve and ensure future usability of data and science 
products are delivered to the DAACs and checked for completeness. 
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Figure 1: NASA Project Life-Cycle [based on NPR 7120.5E (NASA, 2012)] 


3. INTERFACES AND AGREEMENTS 

As indicated above, the nature of Earth science data management implies involvement of several 
organizations working collaboratively for the benefit of the end users. To ensure that responsibilities of the 
different organizations are mutually understood, agreements are developed between them. Figure 2 shows 
the various organizational interfaces needed in support of a satellite mission. The left half of the figure shows 
Mission System components that handle mission operations, data capture and low-level processing 
(telemetry error corrections, overlap removal, etc. to generate level 0 products). The right half of the figure 
shows the Science System components managed by the ESDIS Project. An Inter-Project Agreement (I PA) 
between the ESDIS and the flight project, and a Working Agreement between the ESDIS Project and the 
Science Team spells out the responsibilities of the respective organizations at a high level. The Interface 
Control Documents (ICDs) shown in the figure are detailed documents describing all the interfaces and data 
exchanges that occur between the respective organizations. The Science Teams develop algorithm 
theoretical basis documents (ATBDs), which they publish as well as provide to the science data processing 
systems responsible for producing the data products from the satellite observations. The science data 
processing systems have Interface Requirements Documents (IRD) and Operations Agreements with the 
DAACs. The ESDIS Project describes the list of requirements that the DAACs have to meet in the Archive, 
Distribution and User Services Requirements Document. 

The science program management processes established by NASA ensure that the products generated by 
the science teams associated with satellite missions as well as other types of projects are of high quality and 



well documented, and the interface agreements summarized above are intended to ensure that the 
information is conveyed to the DAACs for archiving and disseminating to the user community. 
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Figure 2. Organizations and Interfaces to Support Satellite Mission Data Management 


4. CAPABILITIES OF EOSDIS 

EOSDIS is a distributed system with several components: Distributed Active Archive Centers (DAACs) 
archive and distribute the Earth science data products, and provide assistance to users in data acquisition, 
search, access, and usage. There are 12 DAACs distributed across the U.S., each of which is specialized in 
a specific set of Earth science disciplines. Science Investigator-led Processing systems (SIPSs), collocated 
with the leaders of the science teams for the respective instruments, generate the data products from the 
observation data from the instruments and transfer the data products to the DAACs for archive and 
distribution. Databases of metadata provide access to users at dataset (collection) level as well as individual 
file (granule) level. NASA recently implemented the Common Metadata Repository (CMR) to unite collection 
level and file level metadata. This provides a source of unified, high-quality and reliable Earth Science 
metadata across NASA data holdings. 

The Earthdata website, http://earthdata.nasa.gov , is a comprehensive, sustainable, and evolvable website 
that provides a unified view of NASA’s Earth science data system resources, and provides links to various 
ways to access data and related content as well as external sites. The Land Atmosphere Near Real-time 
Capability for EOS (LANCE) provides data within three hours of observation for use in time-critical 
applications such as numerical weather prediction; forecasting and monitoring natural hazards, 
ecological/invasive species, agriculture, and air quality; and providing help with disaster response. The 
Global Imagery Browse Services provide a visualization capability to enable users to view several EOSDIS 
datasets in the form of images quickly, at full resolution, and access the parts of interest to them. GIBS is 
supported by WorldView, an open source browser-based client. The ESDIS Metrics System collects 
information on ingest, archive and distribution of all EOSDIS datasets and has the capability to generate 
various summary reports such as volumes and numbers of files distributed to users during a given time 
period. It helps keep track of system loading, usage of various datasets, archive growth, growth in data use 
over time, etc. There are several data tools, developed by the ESDIS Project and the DAACs to help with 
search and order, data handling (e.g., format conversion, data manipulation), subsetting, filtering, 
geolocation, reprojection, visualization and analysis. 

The capabilities of EOSDIS have been evolving throughout its existence. Community inputs and user 
feedback have been essential to the evolution. User Working Groups associated with each of the DAACs 
provide inputs on their data and services. Annual surveys of the user community provide an indication of 
their satisfaction as well as specific comments regarding desired system improvements. NASA has 
established a set of Earth Science Data System Working Groups (ESDSWG) to systematize obtaining 











recommendations from NASA-funded “Community Data System" teams involved in development of 
specialized and innovative services to data users and/or research products offering new scientific insight. 
Members of these teams, working with those from the ESDIS Project and the DAACs, address several topics 
such as handling airborne data, cloud computing, data preservation practices, data quality, data recipes, 
dataset interoperability, provenance for Earth science, search relevance and visualization. Independent 
working groups are formed to address each of the topics. They define problems, perform analyses and 
recommend solutions to be considered for implementation in EOSDIS. 

5. DATA PRESERVATION AND STEWARDSHIP 

The data acquired from various sources (satellites, aircraft, field campaigns, etc.), and scientific products 
derived from them are a very valuable resource, whose utility will last well beyond the lifetime of the projects 
that acquire them. As long as there is active research interest in such data and derived products, NASA is 
committed to archive and distribute them to the user community. Even beyond that period, the data and 
derived products will need to be preserved for the benefit of future generations. Preservation involves long- 
term protection of: 1. Data bits; 2. Discoverability and accessibility; 3. Readability; 4. Understandability; 5. 
Usability; and 6. Reproducibility of results. NASA is taking actions in each of these six areas as described 
below. 

Data bits: While collecting, processing and archiving the data, there are several subsystems within EOSDIS 
among which the data files need to be transferred. To ensure their integrity, checksums are used on a 
regular basis. The data are regularly migrated from older media to newer media as technology evolves and 
older media and readers approach obsolescence. (It should be noted that an extraordinary effort was 
required to recover data from the Nimbus-2 satellite. This was because the data had not been migrated from 
the 1 970’s media into newer media.) Data are backed up in order to avoid catastrophic loss. All of the raw 
(Level 0) data from satellites are held in a separate back-up archive physically distant from the DAACs. The 
data are periodically checked to ensure they are recoverable. The software for generating the higher level 
products from the raw data is held at the DAACs and SIPSs. Given this, it is possible to regenerate the 
higher level products in the event of their loss. While this protects against catastrophic loss, the cost and 
disruption to service are generally not acceptable, especially in the active archive phase of the data lifecycle. 
Therefore raw data and higher level products are backed-up at the DAACs as well. A periodic assessment is 
made of the risk of data loss and impact on users given the back-up approach being used at the DAACs for 
different datasets and the expected time for recovery in case of loss of the primary copy. Corrective actions 
are taken when the risk and impact are considered unacceptable. For example, some datasets backed up at 
a DAAC’s organization may be in a separate building from the DAAC, but this may still be considered risky 
subject to loss due to events such as hurricanes. In such cases the back-ups are moved to a distant location, 
away from the DAAC’s campus. In some cases, the datasets with high user demand are backed up both at a 
distant location and locally at the DAAC to ensure fast recovery in case the primary copy is lost or corrupted. 

Discoverability and Accessibility: Standard metadata are critical for discoverability of data, and they need to 
be preserved in order to maintain discoverability in the long-term. As datasets are produced and archived in 
the DAACs, the processing software automatically generates metadata at the individual file level and the 
metadata repository is constantly populated. Metadata at the collection level are produced to aid in discovery 
of datasets. Historically, the metadata model used by EOSDIS has influenced the Federal Geographic Data 
Committee (FGDC) metadata content standard for remote sensing data as well as the more recent ISO 
metadata standards. Automated generation of metadata has been in place since the beginning of the EOS 
missions. The Common Metadata Repository, mentioned in Section 4 above, is a significant step in 
improving the consistency of metadata, and compatibility with the current ISO 19115 standards. There are 
various search and order tools developed by the ESDIS Project and the DAACs that facilitate discovery and 
access by providing multiple paths to datasets. A significant improvement in discoverability has resulted from 
the recently developed search capability on the website http://earthdata.nasa.gov . The accessibility to data 
improved significantly after the data were migrated from the near-line robotic tape libraries into on-line disk 
archives during 2005-2008. NASA is assigning Digital Object Identifiers to all the datasets in EOSDIS with 
standardized content on their landing pages. This will ensure that the datasets will have unique and 
persistent identifiers, that they are citable and also enhance discovery and access to the data. 


Readability: In order to preserve readability, it is essential to maintain the knowledge about the data formats, 
and possibly the software used to read the data. EOSDIS supports a number of data formats that are in use 
by the Earth science community. The Hierarchical Data Format (HDF) is the primary format in which most of 
the datasets in EOSDIS are generated and archived, even though there are several datasets in other 
formats as well. EOSDIS supports translations into other formats such as NetCDF, GeoTIFF and binary, 
upon request by users. The HDF is in reality a self-documenting formatting system, which provides a flexible 
structure within which a data producer can define a profile and generate a set of files. The HDF library 
consists of software for reading and writing files according to the defined profile within the HDF structure. 

The flexibility in structure is an advantage from the point of view of the data producer. However, from the 
point of view of a future user, it becomes necessary either to maintain the version of the software library that 
created the datasets or document the structure of the dataset such that software to read the data can be 
easily created. During the current active archive phase, the HDF libraries are being maintained. To prepare 
for the future, the HDF Archive Mapper has been developed. This mapper generates an XML file recording 
the “Layout Map” of a given dataset, which can be used to generate read software easily. This obviates the 
need for maintaining HDF libraries indefinitely. Efforts are also being made to increase consistency among 
the datasets by facilitating use of common structures across groups of datasets. For example, a common 
understanding about the structure and metadata of products within the HDF structure was developed among 
the instrument teams prior to the launch of the EOS Aura satellite in 2004. More recently, an HDF Product 
Designer has been developed, which enables NASA Earth Science projects, and others, to design 
interoperable HDF products that are compliant with community conventions, and to share those designs 
across teams. It imports existing file structures and metadata to a variety of formats. It outputs designs and 
computer programs for writing products in various programming languages. 

Understandability: To preserve understandability, it is essential to maintain documentation associated with 
the data products. Algorithm Theoretical Basis Documents (ATBDs) for each of the products are developed 
by the scientists responsible for generating data products. While these are generally not detailed enough to 
reproduce the products, they are sufficient for users to understand the theory behind them. The DAACs 
maintain product information pages, guides, answers to frequently asked questions (FAQs), and in some 
cases, forums for exchange of information to help users understand data products. During the active 
archive phase, the science teams generally hold and maintain the ATBDs, and the DAACs maintain the other 
artifacts mentioned above. For long term preservation these need to be gathered to ensure that all relevant 
materials remain conveniently available. 

Usability: While readability and understandability are prerequisites for usability, information is also needed 
about whether a dataset is suitable and accurate enough for the user’s purposes. The accuracy 
assessments, documentation about data quality, and validation efforts carried out by the science teams 
responsible for producing the datasets provide such information. Some of this information is held by the 
science teams, while other documentation is included with the data archived at the DAACs. The materials 
needed for long term preservation will need to be identified and collected to ensure that they will be available 
along with the datasets. 

Reproducibility of results: In order to enable reproducibility of results, e.g., scientific conclusions published 
using a given version of a given dataset in the future, it is essential to preserve the source code and/or 
software specification documents that precisely describe the computations performed in a given version of 
the dataset, as well as the inputs used such as calibration files and ancillary data. There may be many 
versions of datasets during the course of a mission, and it may not be practical to preserve all of them. 
However, it is important to preserve the versions of datasets or the means of regenerating them when they 
result in peer-reviewed publications. 

Especially to preserve understandability, usability and reproducibility, it is necessary that all the metadata, 
documentation, software, and ancillary data required are identified and collected before the end of the 
respective projects. In 2011, NASA developed the Earth Science Data Preservation Content Specification 
(NASA, 2011). This document identifies eight categories of items to be preserved: Pre-flight/Pre-Operations 
Calibration, Science Data Products, Science Data Product Documentation, Mission Data Calibration, 

Science Data Product Software, Science Data Product Algorithm Input, Science Data Product Validation and 
Science Data Software Tools. Several content items in each of these categories have been identified, along 
with the rationale for preserving them. This specification levies requirements on current and future projects. 

It is also used as a checklist to gather preservation content on a “best efforts” basis from earlier projects that 



had been in operation prior to the development of the specification. For example, it has been used in the 
case of the Geoscience Laser Altimeter System (GLAS) instrument on ICESat, the High Resolution Dynamic 
Limb Sounder (HIRDLS) and Microwave Limb Sounder (MLS) instruments on the EOS Aura satellite, and 
Total Ozone Measuring System - Earth Probe (TOMS-EP). It is important to plan for preservation starting 
early in the project lifecycle. For instance, in the case of HIRDLS and GLAS, there was a significant effort 
over a period of several months during their closeout activities to identify, gather and deliver the relevant 
preservation content to the DAACs. The instrument teams had to review thousands of documents to ensure 
that the appropriate documents needed to support the datasets were collected and delivered. To be efficient 
and effective, preservation content items should be collected before the individuals knowledgeable about the 
items complete their contribution to the project and move on to other activities. 

In order to stay abreast of current standards and technology, and in some cases influence standards, ESDIS 
Project and other EOSDIS personnel are active in a variety of internal, interagency and international working 
groups. 


Internal Working Groups 

Within the ESDIS-sponsored Earth Science Data Systems Working Groups (ESDSWG), the Data 
Stewardship Interest Area consists of Working Groups (WG’s) covering the following topics: 

• Citations and Identifiers WG. The ESDIS Project has been assigning Digital Object Identifiers (DOI's) 
to EOSDIS datasets held at the DAACs. This working group has examined the assignment process 
and made suggestions for improvement, as well as recommended format and contents of landing 
pages associated with the DOI's. This working group is currently examining the need for assignment 
of persistent, unique identifiers for objects other than datasets in order to ensure understandability 
and usability of datasets and easy access to dataset-related information. 

• Dataset Interoperability WG, analyzes issues with data formats of products offered by EOSDIS, and 
makes suggestions for changes to ensure that the products are interoperable with a broader set of 
tools extant in the community. An example of this is a set of suggestions for formatting datasets in 
the Hierarchical Data Format (HDF) so that they can be easily read by Network Common Data Form 
(NetCDF) tools. 

• Data Preservation Practices WG has developed a document titled Data Preservation Guidelines 
(DPG) which complements the Preservation Content Specification (PCS). While the PCS covers 
“what” needs to be preserved, the DPG has recommendations on “when” and “how”. 

• Data Quality V\IG has considered how information on the quality of data products are being collected 
and conveyed to users and made recommendations on how the process can be improved to best 
benefit users. This working group is currently examining the complexity of implementation of such 
recommendations across EOSDIS. 

• Provenance for Earth Science WG has extended the World-Wide Web Consortium’s (W3C) PROV 
standard for representation of provenance to Earth science, and has implemented working 
prototypes to generate lineage graphs of product generation as well as links to relevant artifacts. 

Interagency and International Groups 

NASA also actively collaborates in interagency and international organizations. The Federation of Earth 
Science Information Partners (ESIP) is a group with over 150 member organizations consisting of serval U.S. 
agencies, university and commercial entities. NASA representatives participate in and lead many activities 
within the ESIP. In particular, they play an active role in the ESIP Data Stewardship Committee. As result of 
this participation, the emerging Provenance and Context Content Standard (PCCS) originated in 201 1 
(Ramapriyan et al, 2012), based on which NASA’s PCS document was developed. The ESDIS Project has 
been active in the CEOS Working Group on Information Systems and Services (WGISS) since its start. 
(CEOS is the satellite arm of the Group on Earth Observations (GEO) supporting development of GEOSS.) 
The CEOS WGISS promotes collaboration in the development of systems and services that manage and 
supply Earth observing data from its members. The activities and expertise of WGISS span the full range of 
the information life cycle , starting with “definition of requirements and metadata for the initial ingestion of 
satellite data into archives through to the incorporation of derived information into end-user applications”. The 
International Directory Network (IDN) is part of NASA's contribution to CEOS and other international Earth 



sciences efforts. WGISS is developing the CEOS WGISS Integrated Catalog (CWIC) to allow for easier 
search and access of EO data via partnering of CEOS agency data systems. 

6. SUMMARY AND CONCLUSIONS 

NASA has been collecting Earth observation data for over 50 years using instruments on board satellites, 
aircraft and ground-based systems. With the inception of the EOS Program in 1990, NASA established the 
ESDIS Project and initiated development of the EOSDIS. With its 12 Distributed Active Archive Centers, 
EOSDIS is responsible for the stewardship of most of NASA's Earth science data. The data stewardship 
responsibilities include ensuring that the data and information content are reliable, of high quality, easily 
accessible, and usable for as long as they are considered to be of value. To meet these responsibilities, the 
ESDIS Project coordinates with the data producing missions or projects throughout their lifecycles. NASA 
ensures that the active archive capabilities are available as long as there is interest in using the data, which 
is well beyond the life time of missions. In 201 1 , NASA developed the Earth Science Data Preservation 
Content Specifications, which is being used to ensure that the data and all the associated metadata and 
documentation from missions are collected and archived in preparation for permanent preservation, so that 
future generations will still be able to understand and uses the data products from today’s missions. NASA’s 
approach to data preservation and stewardship is intended to provide for long-term protection of 1. Data 
bits; 2. Discoverability and accessibility; 3. Readability; 4. Understandability; 5. Usability; and 6. 
Reproducibility of results. 
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